1. Field of the Invention
The present invention relates to an image processing method for processing images in real-time, an image processing method for generating an image data structure suitable for processing images in real-time, and a recording medium recording the image data structure or image processing program concerned.
2. Description of the Related Art
In a home video game machine, moving pictures of objects of a game are created in real-time synchronizing with the progress of the game. For the image processing in such a game machine, it is necessary to move the positions of the objects in the game responding to the input control signals of the operator within a short period, a frame period for example, and to draw images corresponding to the movement positions in the image memory (frame memory).
In order to express movement more naturally, it is necessary that characters, e.g. living creatures, as the objects of the game can move joint parts and muscle parts along with the movement of limbs and necks, and that the joint part becomes a different shape according to the movement of other polygons. In other words, in order to express more natural movement, the part between the upper arm and shoulder of an individual or the part between the upper arm and lower arm requires such movement as raising the muscles of the upper arm and shoulder depending on the movement of the polygons.
A drawing method for giving natural movement to such joint parts specifies the vertices to be connected for models on both sides of the joint, and adds a polygon for the joint between the vertices. With such a drawing method, however, the joint part is drawn by standardized polygons, where movement corresponding to the change of angle cannot be expressed, and as a result pictures to be generated appear unnatural.
In order to draw more natural movement, it is proposed to utilize integrated shaping models called xe2x80x9cenvelopesxe2x80x9d using three dimensional modelers in the field of image processing of movies such as CG movies (movies using computer graphics), which do not require real-time processing. In these integrated shaping models, objects (characters) are comprised of polygons and bones which influence the polygons. In other words, an integrated shaping model is comprised of polygons, which are the outer surface of the character and are drawn based on actual movement, and bones, which influence the positions of the polygons with a predetermined weight. New positions of bones which influence the positions of the polygons are computed by a position conversion matrix determined according to the operation input of the operator, and the positions of the vertices of the polygons are determined according to the degree of influence (weight) from the bones. Complicated movements can be expressed by polygons by executing rendering (drawing) on the polygons based on the determined vertices.
FIG. 1 is a drawing depicting an example of movement of the above mentioned integrated shaping model in a CG movie. The model shown in FIG. 1 is a model comprised of an arm part 1 and a hand part 2, and FIG. 1A shows the basic form of the model. When the arm is bent such that the upper arm 3 of the arm part 1 becomes more vertical and the lower arm part 4 of the arm part 1 becomes more horizontal from the basic form, the state shown in FIG. 1B is expected. In other words, the joint part connecting the upper arm 3 and the lower arm part 4 exhibit a natural bent shape and the inner muscles 3A of the upper arm 3 tense up.
FIG. 2 is an explanatory drawing depicting conventional image processing using an integrated shaping model in CG movies. FIG. 3 is a flow chart depicting the image processing procedure thereof. FIG. 2 shows the data structure 620 of the integrated shaping model (envelope). In this example, a total of seven models, models 1-7, are in the tree type hierarchical structure shown in FIG. 2. The data structure of each model is the same as the data 624 of the model 4 and the data 627 of the model 7.
When a model has polygons to be displayed, the model has vertices which are the composing elements of the polygons. Therefore the data structure 624 of the model 4 includes the vertex list 632 and the polygon list 635 of the polygons comprised of a combination of the vertices. The vertex list 632 includes position information and normal line data of each vertex in model local coordinates. Each model also has a model matrix 631, which is information denoting relative position with respect to a model in the higher hierarchy. The model 4, for example, has rotation, shift (translation) and size (scale) as the relative position information with respect to model 3. The model matrix 631 comprised of such position information, can also be matrix data when the coordinate system of model 3 is converted to the coordinate system of model 4.
The data structure 624 of the model 4 also has the weight list 633 including the vertices of other models which model 4 influences, and the weight values, which indicate the degree of that influence. In the case of the example in FIG. 1, for example, the model of the lower arm 4 influences the muscle part 3A of the upper arm 3 with a predetermined weight value. Therefore in the weight list of the model of the lower arm part 4, the ID of the model of the upper arm 3, the indexes of the vertices thereof (vertex IDs) and the weight values are included.
If the model 4 of the model structure in FIG. 2 is the lower arm part 4 in FIG. 1, and the model 7 is the upper arm part 3, then the positions of vertices of the muscle part 3A in the upper arm part 3 model change, influenced by the movement of the upper arm part 3 and the movement of the lower arm part 4, and the image where the muscle part 3A rises is drawn. Now the image processing procedure by a three dimensional modeler, which is generally used for CG movies, will be explained according to FIG. 3. At first, the matrix for implementing the movements of all models is calculated and saved in the data structure of each model (Step 602). Then referring to the weight list 633, vertex coordinates and normal line data (vertex data) indicated by the vertex indexes (vertex IDs) of the model (model 7) which is influenced by the target model (model 4) are read from the vertex list 632 of the model 7 (Step 603), the read vertex data is converted to the vertex data when the vertex data belongs to the local coordinate system of the model 4 based on the relationship in the basic form, and the converted vertex coordinates and normal line data are multiplied by the model matrix of the model 4 (Step 604). Also the vertex coordinates and the normal line data multiplied by the model matrix are multiplied by the weight value in the weight list 633, and the values are integrated with the values in the weight vertex list 634 in the data structure 627 of the model 7.
The above procedures 603, 604 and 605 are executed for the entire weight list (Step 606) and are executed for all models (Step 607). As a result, coordinate conversion corresponding to the movement of characters at actual vertices of all models, and the position movement according to the degree of influence from other models are executed. Then the normal line vectors of the weight vertex list of all models are normalized so that the normal vectors can be used for light source processing in rendering, which is executed later (Step 608). And referring to the vertex positions in the weight vertex list, the drawing (rendering) of the polygons in the polygon list is executed (Step 609).
For the above operations, the matrix operation and weight operation are executed for the vertices of all models, and rendering processing for the polygons is executed after the above operations complete. Or, one vertex is focused on and the matrix operation and weight operation of models which influence the vertex are executed, then the new vertex coordinate of the vertex is determined, and final rendering processing is executed.
This image processing method for CG movies, however, cannot be applied to high-speed image processing executed in real-time in a game machine. When the matrix operation and weight operation are executed for the vertices of all models, the operations must be executed with reference to the vertex lists and model matrixes of different models in the data structure of the model. If such operations are executed involving a reference procedure using a pointer in the hierarchical structure of the model data distributed in memory, the pointer must be changed frequently. There are shortcomings to executing such operations in a game machine, since the working efficiency of the cache memory disposed in the CPU of the game machine is poor. Also when the matrix operation and the weight operation are executed for one vertex focusing on the vertex, the buffer area for the weight operation must be secured for all vertices. If this operation is executed in a simple-structured game machine, as opposed to a supercomputer having a large capacity memory and a high-speed processing capacity, the memory area required will be enormous, and real-time processing is difficult, making operation impractical.
With the foregoing in view, it is an object of the present invention to provide an image processing method which allows image processing in real-time utilizing integration shaping models, an image processing method for generating image data on integrated shaping models used for the image processing thereof, and a recording medium recording the image data and the image processing program concerned.
It is another object of the present invention to provide an image processing method based on a converter program for generating image data, which allows image processing in real-time utilizing integrated shaping models, from three dimensional model data.
It is still another object of the present invention to provide an image processing method for transforming joint parts into a natural shape according to the movement of a character, a recording medium recording an image processing program to execute the image processing method and a recording medium recording data having a structure suitable for the image processing.
To achieve the above objectives, the present invention is a computer-readable recording medium recording integrated shaping model data having a plurality of models linked by a hierarchical structure where at least a first model has a plurality of vertices constituting polygons, and at least the position of a first vertex is influenced by positions of a plurality of models and weight values from these models, comprising: format data of a common vertex buffer for saving the data on the plurality of vertices in the plurality of models for each model; a vertex list which is created for each one of the models which influence the vertices and has vertex data specified by a vertex ID in the common vertex buffer; model matrix data which is created for each one of the models and sets the model positions of the plurality of models in basic forms; and a polygon list which is created for each one of the models having the polygons and has polygon data where the vertex ID is attribute data, wherein the vertex data in the vertex list has at least the position data of the vertex, a weight value from the model where the vertex list belongs, and vertex ID in the common vertex buffer corresponding to the vertex.
The above recording medium recording the integrated shaping model data allows image processing required for drawing models in real-time in a game machine.
The present invention is also the above mentioned invention, wherein the first vertex influenced by the plurality of models have distributed vertex data for the number of the plurality of models, and the distributed vertex data belongs to the vertex lists of the models which influence the first vertex.
The present invention is also the above mentioned invention, wherein the vertex list is divided into start weight vertices, for which the vertex operation is executed first when tracing is executed according to the hierarchical structure of the models, middle weight vertices, for which the vertex operation is not executed first or last, and end weight vertices, for which the vertex operation is executed last.
The present invention is also the above invention, wherein the vertex list further has data on native vertices which are influenced only by the model which this vertex list belongs to.
The present invention is also the above invention, wherein the vertex data has coordinate data and normal line data of the vertices.
The present invention is also the above invention, wherein the vertex data follows a local coordinate system of the model of the vertex list which the vertex data belongs to.
The present invention is also the above invention, wherein the vertex ID in the common vertex buffer has an entry number of the vertex of each model and the offset value corresponding to the number of vertices of each model.
The present invention is also the above invention, wherein the polygon list has a drawing command which instructs the drawing of the polygon of a predetermined model, or a non-drawing command which instructs not to draw the polygon of the model which this polygon list belongs to.
To achieve the above objectives, the present invention is a computer-readable recording medium recording integrated shaping model data, having a plurality of models linked by a hierarchical structure where at least a first model has a plurality of vertices constituting polygons, and at least a first vertex position is influenced by positions of a plurality of models and weight values from these models, comprising: format data of a common vertex buffer for saving the data on the plurality of vertices in the plurality of models for each model; a vertex list which is created for each one of the models which influence the above vertices and has vertex data specified by a vertex ID in the common vertex buffer; and a polygon list which is created for each one of the models having the polygons and has the polygon data where the vertex ID is attribute data.
To achieve the above objectives, the present invention is an image processing method for converting original model data into integrated shaping model data having a plurality of models linked by a hierarchical structure where at least a first model has a plurality of vertices constituting polygons, and at least the position of a first vertex is influenced by positions of a plurality of models and weight values from these models, wherein the original model data has a plurality of model data linked by the hierarchical structure, the original model data comprises an original vertex list having vertex data of this model, a weight list having vertex data on vertices which this model influences, and an original polygon list where the vertex ID of the vertex list is attribute data, and the above image processing method comprises the steps of: generating format data of a common vertex buffer to store data on the plurality of vertices in the plurality of models for each model; generating a vertex list which is created for each model which influences the vertex, and has vertex data specified by a vertex ID in the common vertex buffer; and generating a polygon list which is created for each model having the polygons where the vertex ID is attribute data.
According to the above image processing method, conversion data for an integrated shaping model which allows image processing in real-time can be generated.
The present invention is also the above image processing method, wherein the vertex data in the vertex list has at least the position data of the vertices, weight values from the model which the vertex list belongs to, and vertex IDs in the common vertex buffer corresponding to these vertices.
The present invention is also the above image processing method, wherein the step for generating the vertex list generates distributed vertices for the number of the plurality of models for the first vertex which is influenced by the plurality of models, and distributes the data on the distributed vertices to the vertex list of the models which influence the first vertex.
The present invention is also the above image processing method, wherein the step for generating the vertex list divides the vertex data into data for start weight vertices, for which the vertex operation is executed first when tracing is executed according to the hierarchical structure of the models, data for middle weight vertices, for which the vertex operation is not executed first or last, and data for end weight vertices, for which the vertex operation is executed last, in the vertex list.
The present invention is also the above image processing method, where the vertex ID has an entry number of the vertex in each model, and an offset value corresponding to the number of vertices of each model.
The present invention is also the above image processing method, wherein the above image processing method further has a drawing command generation step where: a dummy vertex operation is executed for the vertex data in the vertex list of the model according to the sequence of tracing following the hierarchical structure of the models; when the dummy vertex operation for the vertex list of one model ends, a non-drawing command is generated in the polygon list of this model if the weight calculation for the vertices of this model has not ended; and a drawing command for the non-drawing model is generated in the polygon list of the target model of the dummy vertex operation if the weight calculation for the vertices of the non-drawing model ends.
The present invention is also the above image processing method, wherein the above image processing method further has a one weight vertex list generation step, where the first vertex which is influenced by the plurality of models is converted to a vertex which is influenced only by a model having the largest degree of influence among the plurality of models, and the data on the converted vertex is distributed to the vertex list of this model.
To achieve the above objectives, the present invention is an image processing method for drawing an integrated shaping model which has a plurality of models linked by a hierarchical structure, where at least a first model has a plurality of vertices constituting polygons and at least the position of a first vertex is influenced by positions of a plurality of models and weight values from these models, wherein the data of the integrated shaping model comprises: format data of a common vertex buffer which stores data on the plurality of vertices in the plurality of models for each model; a vertex list which is created for each model which influences the vertices and has vertex data specified by a vertex ID in the common vertex buffer; and a polygon list which is created for each model having the polygons and includes polygon data where the vertex ID is attribute data, the image processing method comprising steps of: generating the common vertex buffer corresponding to the plurality of models in the sequence of tracing of the hierarchical structure according to the format data; generating model matrix data where the positions of the models are set based on the game progress data; generating common vertex data by executing the matrix calculation for generating vertex data after movement according to the model matrix data and weight calculation for integrating the weight values from the models to the vertex data after movement, for the vertex data of the vertex list of the models, and by storing or adding this operated vertex data to areas according to the vertex IDs in the common vertex buffer; and rendering the polygons according to the common vertex data.
According to the above image processing method, characters of the integrated shaping models can be drawn in real-time.
The present invention is also the above image processing method, wherein the vertex data in the vertex list has at least the position data of the vertex, weight values from the model which the vertex list belongs to, and the vertex IDs in the common vertex buffer corresponding to these vertices, and in the step of generating the common vertex data, the position data is multiplied by the model matrix for the above matrix calculation, and the above operated position data is multiplied by the weight value for the above weight calculation.
The present invention is also the above image processing method, wherein the vertex list is divided into start weight vertices, for which the vertex operation is executed first when tracing is executed according to the hierarchical structure of the models, middle weight vertices, for which the vertex operation is not executed first or last, and end weight vertices, for which the vertex operation is executed last, and in the step of generating the common vertex data, the operated vertex data for the start weight vertices is stored in areas corresponding to the vertex IDs in the common vertex buffer, and the operated vertex data for the middle weight vertices and end weight vertices are added to the vertex data in areas corresponding to the vertex IDs in the common vertex buffer.
The present invention is also the above image processing method, wherein the above polygon list has a drawing command instructing to draw the polygons of a predetermined model or a non-drawing command instructing not to draw the polygons of the model which this polygon list belongs to, and the above rendering step refers to the polygon list of the model each time the above common vertex data generation step of each model ends, and executes the above rendering according to the drawing command.
The present invention is also the above image processing method, further comprising a step for releasing the area of the common vertex buffer corresponding to the model for which the above rendering is executed.
The present invention is also the above image processing method, wherein the above model data also has a vertex list for one weight where the first vertex which influenced by the plurality of models is converted to a vertex which is influenced only by the model which degree of influence is largest among the plurality of models, and data of the vertex is distributed in the vertex list of this model, and the above image processing method generates the common vertex data for a predetermined model according to the vertex list for one weight.
To achieve the above objectives, the present invention provides a recording medium recording a program for a computer to execute the above mentioned image processing method. With this recording medium, characters of an integrated shaping model can be drawn in real-time in game machines.